Synthesize the week, justify every decision, and design how you'd watch the pipeline's quality in real time — the bridge into Week 9
Day 40~60 minutesReview
Day 40 of 66
What you can now do
One week into Phase 2 and your altitude has changed for good. You no longer just label what lands in your queue — you can design the queue, the flow it feeds, and the measurements that prove it works. Say this back without looking:
The Week 8 spine
A pipeline is a flow: raw → sampling → queue → routing → consensus → review → final label → training set.
Stratified sampling guarantees coverage and lets you oversample weak spots — what you sample is what you teach.
Consensus scales with cost-of-error: N = 1 easy, N = 3 majority hard/safety; more consensus buys quality and spends throughput.
Review tiers escalate only the exceptions — splits, gold misses, high-stakes, flagged-ambiguous.
The tool (Label Studio, Argilla) executes the design; the design is the skill.
Defend every decision
A pipeline you can't justify is a pipeline you'll lose the moment someone with budget pushes back. For your own design, have a one-sentence defense ready for each dial — stated as a trade-off, not a preference.
Why this sampling? "Stratified by category so no slice is starved, oversampling text-rendering 2× because it's our measured weak spot."
Why this consensus split? "N = 1 where an error is cheap, N = 3 where it's expensive — the same quality as flat 3-way at roughly half the cost."
Why this gold rate? "~10% — enough gold hits to catch a drifting annotator within a week, cheap enough not to crowd out new labels."
Why this escalation rule? "Only splits, gold misses, and high-stakes items, because escalating everything makes reviewers the bottleneck."
Self-quiz (no notes)
Answer out loud or in writing. If any answer is shaky, reread that day before Week 9.
Name the stages of an annotation pipeline in order, from raw item to training set.
You have a fixed annotator budget. Why is a mixed N=1/N=3 design usually better than putting everything on 3-way consensus?
What is gold for, and what single step makes it actually work?
Give two triggers that should escalate a label to a review tier — and explain why you don't escalate everything.
A teammate says "let's just buy Label Studio and we're done." What's your one-sentence correction?
Answer key (peek after you try)
1) raw items → sampling → queue → routing → consensus → review → final label → training set. 2) Triple consensus on easy items spends budget for quality you didn't need; concentrating the 3 eyes on hard/safety items gets the same quality where it matters at roughly half the cost. 3) Gold (known-answer items) measures quality live, per annotator; it only works if it's shuffled in invisibly so annotators can't treat it specially. 4) Any two of: a no-majority consensus split, a recent gold miss, a high-stakes/safety item, an annotator-flagged ambiguity. You don't escalate everything because reviewers would become the bottleneck and throughput would collapse. 5) "The tool just executes the design — consensus, gold rate, and review tiers are decisions we still have to make, and those are what determine the data's quality."
Instrument it for live quality — the bridge to Week 9
A pipeline you build once is a project. A pipeline you watch is an operation. The last move this week is to design how you'd know, continuously, whether the data is getting better or worse — which is exactly the discipline Week 9 turns into a daily heartbeat.
How you'd instrument it
Gold accuracy over time. Track rolling accuracy on gold, per annotator and overall. A drop is your earliest warning that quality is slipping.
Agreement rate on N = 3 items. Rising disagreement means the task got harder or the guideline aged — a guideline signal, not a people one (Week 3).
Escalation rate. A spike in items hitting review means something upstream changed — new annotators, a shifted item distribution, an unclear rule.
Queue composition drift. Watch that your strata stay balanced as the raw pool shifts, so the model keeps learning where you intended.
Up next — Week 9: Quality Operations at Scale
You'll turn these instruments into a real audit: sample labels against gold, compute an accuracy rate with a confidence interval, and detect drift week over week — the operational loop that keeps a pipeline honest after it's live. The pipeline you designed this week is the thing Week 9 keeps alive.
Close the week
Wrap-up
Before Week 9
Self-quiz answers all solid (or reread the weak day)
Pipeline design documented: sampling, consensus, gold, review tiers — each with a one-line defense
pipeline.py runs on a task you actually do, with composition reported
The four instrumentation signals noted, ready to build in Week 9
The lead move
A contributor can run a pipeline. A lead can defend every dial as a trade-off when someone with budget pushes back — and then designs the instruments (gold accuracy, agreement, escalation rate, queue drift) that prove the data is staying good after launch.
Say this in an interview:"I don't just build a pipeline and walk away — I have a one-sentence trade-off defense for every decision in it, and I instrument it so I'd see quality slipping from the gold curve before it ever reaches the training set."
Week 8 Takeaways
You can design an annotation pipeline end to end and defend every decision as a trade-off.
You can prototype the queue in code — stratify, consensus by difficulty, hidden gold — and report its composition.
You know where the pipeline fails and how to escalate the exceptions without drowning your reviewers.
You can instrument it for live quality — the exact setup Week 9 turns into a running quality operation.